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Abstract: Promoting blog content is a social activity — a means of communicating one student's 
appreciation of another student's work. This article explores the feasibility of using student 
promotions of content, in a blogosphere, to identify quality content, and implications for 
students and instructors. We show that students actively and voluntarily promote content, 
identify quality material with considerable accuracy, and use promotion data to select what to 
read. We explore the benefits of knowing which students are good and poor predictors of quality 
content, and what instructors can do with this information in terms of feedback and guidance. 

Keywords: Blogging, social blogging, liking, promoting, assessment, quality content, learning 
analytics, knowledge community, social recommendation, recommender system 


1 INTRODUCTION 

The traditional classroom is changing, with devices (laptop, smartphone, et cetera) that access the 
Internet, connecting the classroom to the larger world. For some classes, all communication is online, 
making the classroom completely digital. Other classes are blended to some degree or another — part 
classroom, part online. Using digital communication is relatively new in learning. Neither learners nor 
instructors can be assumed to pick up this new learning tool automatically and fully understand its 
possibilities or use it effectively. To complicate matters further, the technology is rapidly changing; new 
applications are constantly being created and devices upgraded. What does not change is that, with each 
day, the integration of learning with digital technologies is further advanced. This marriage produces 
significant amounts of data. This is where the field of learning analytics comes into play, applying 
analytical methods to the content created by the learners online (Siemens, 2011). 

In class, the instructor has face-to-face access to her students. Online, students and instructors become 
entities of the Internet, where their presentation of self is constrained by the selected mode of 
communication. Common communication methods are email, forum posts, blog posts, and chat rooms 
(possibly with audio and video). These methods of communication have different features than face-to- 
face communication; each of them providing alternate constraints and opportunities for contributing to 
a learning discourse (e.g., Baker, Hansen, Joiner, & Traum, 1999). Each contribution can be bookmarked, 
shared, liked, read multiple times, tagged, referenced, and quoted. Within the bounds of a semester- 
length class of students, the creation of this kind of information is a sign of social activity among 
students. Making visible this kind of social data to the entire class provides important peer feedback and 
recognition; it also potentially creates valuable data for learning analytics. 

This article will show that social data is a significant element in loosely coupled online collaborative 
environments, beneficial to both students and instructors. We will focus on one sort of social data: 
promotions — one student anonymously and subjectively assessing the quality of another student's 
work by clicking a button. The promotion activity can take alternate forms; for example, liking or giving a 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


126 


JOURNAL OF LEARNING ANALYTICS 

(2014). Peer Promotions as a Method to Identify Quality Content. Journal of Learning Analytics, 1(2), 126-150. 


S • LAR 


merit badge. We show that the data can be used to solve significant problems for both students and 
instructors. 


The field of learning analytics applies methodological scrutiny to learner-created content, such as the 
data created through promotions. In online learning settings, collecting data is easy; all activities can be 
saved for analysis. It would not be so easy to get the same data from a classroom discussion. This easy 
access to data has created the field of learning analytics and the data has inspired much new research. 
Creating sense-making visualizations for instructors and administrators (Dawson, 2009), exploring 
knowledge building in online social settings (Ferguson & Buckingham Shum, 2012), and data mining 
institution-sized data sets (Arnold, 2010; Smolin & Budakov, 2012) are examples of work done in this 
field. 

In a technology-mediated environment, the intersection of the learning framework and the analytic 
methods form a middle space (Suthers & Verbert, 2013). An underlying theory of learning is a significant 
element of the foundation of any online learning environment: it provides both a rationale for the design 
of the learning environment, including the scaffolding, and a basis for analysis (p. 2). In other words, 
learning theory motivates the invention of technology, the use of the technology produces data, the 
analysis of the data can suggest methods to fix problems and enhance the learning experience for any 
(or all) of the stakeholders, most notably either the students or the instructor. These enhancements and 
solutions can then again impact the theory and produce another cycle of new problems and solutions. 

Contrast, for example, the underlying assumptions of two alternate projects to support student online 
learning. In one case, the VirtualMathTeam project (Stahl, 2009), students work together online, in small 
groups, at the same time from different locations in a partial joint-focus space; learning for relatively 
short periods of time, by negotiating meaning in the social world (Stahl, Koschmann, & Suthers, 2006). In 
a contrasting case, students collaborate in a blogging environment throughout the semester — working 
together in a loosely coordinated fashion at different times and different locations — creating and 
distributing knowledge, and collaboratively building skills (Alterman & Larusson, 2013). Because the 
learning framework for these two examples is different, the middle space will be different. The analytical 
methods might have some overlap, but some of the goals, results, and conclusions will be radically 
different. 

The research reported on in this article analyzes learner-created content in two classes (107 and 50 
students). The classes were both blended courses, with lectures twice per week, and ongoing 
participation in an online knowledge community as a requirement. While the amount of data is not as 
much as for institution-sized data sets or MOOCs (massively open online courses), the data is still 
significant because it represents the normal conditions for many learning environments. Despite the 
limits of the dataset size, important results can nevertheless be obtained. 

In both classes presented, each as a case study, the students blogged throughout the semester, 
collectively producing much content each week; in one of the classes, students produced roughly 300 
contributions per week totalling around 100,000 words, or ten times the length of this article! Without 
some learning analytic intervention, this amount of raw material for the students to navigate, and for the 
instructor to assess, is overwhelming. 

This article explores the value of promotions in a collaborative online learning environment in a blended 
course. The students work in a loosely coordinated fashion — they work in a blogosphere — producing 
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large amounts of content in a student-owned space. At least two issues emerge because of the number 
of posts; assessment can become overwhelming for the instructor, and finding the relevant content for 
the students. Promotions, as a feature, have the virtue of being peer-created content that does not 
increase the instructor's workload to produce. The issue concerns the value of promotion data: Does the 
promotion activity identify high quality with minimal instructor involvement? 


2 PROMOTIONS AS A SOCIAL ACTIVITY 


In a traditional classroom, a student typically only has access to verified high quality content like books, 
articles, and lectures supplied by the instructor. Online there might be videos, more articles, and 
tutorials by other professionals. Other sources of content are peers, but here the quality of the content 
is mixed. Offline peer content can be produced by study groups, group projects, and classroom 
discussions. Online peer content can be produced in a knowledge community that uses technologies to 
enable content creation and distribution online (e.g., wikis, blogs, forums, and chats). The higher quality 
peer-created content can serve as a valuable resource for learning, but traditionally separating the good 
from the bad is the province of the instructor, which, for large amounts of content, makes it prohibitively 
time-consuming and thus a prime target for learning analytics. 


Giving students access to each other's contributions is the basis for a knowledge community 
(Scardamalia & Bereiter, 1994), with each contribution representing a different, but valid, viewpoint that 
provides scaffolding for others to build upon and improve. Individual perspectives are shared, developed 
(Stahl, 2003), and composed (Suthers, 2006). Different contributions and examples on the same subject 
are multiple representations, which are valuable for learning (Ainsworth, 2006), and can lead to higher 
order thinking (Ellison & Wu, 2008; Philip & Nicholls, 2009). Depending on the technology that mediates 
peer collaboration, there will be different benefits. For example, a blogosphere is especially valuable as a 
resource for co-reflection among the students, which is very helpful for learning (Deng & Yuen, 2011). 

Students produce content of mixed quality; ideally, the best content is foregrounded in some way. One 
solution is to have the instructor and graders identify the quality content for the rest of the class. 
However, as the size of the class increases, the cost of finding good content, without any automated 
assistance, becomes quite labour intensive — perhaps prohibitively so. An alternate solution is to have 
students themselves locate and label important content as they read. In the studies presented in this 
paper, students can attach likes and badges to each other's contributions. Collectively these features can 
be called promotions: the reader considers the content noteworthy and wants to label it publicly as such 
for the community. If the social data is reliable, quality examples will be identified, which can 
subsequently be used as a basis for comparison and self-assessment by other students. This scheme 
potentially makes the large quantity of content being produced more manageable for the students. It 
also can provide valuable feedback, as it potentially both recognizes and confirms the value of an 
individual contribution. 


In this article, we present a social feature (promotions) to filter for quality content. The purpose of this 
feature is to have students collectively identify content that is noteworthy and helpful when creating 
new material. These promotions are anonymous and persistent, making them very useful for later 
exploration by assisting navigation. The instructor can also use the promotions to classify students based 
on their reading and writing abilities. This leads to a more accurate sorting of the quality content, 
beneficial for both the students and the instructor. The main result is that analysis of the learner-created 
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content verifies that student promotions can be used as a basis for identifying high quality material in 
the blogosphere. 


The findings are several-fold: 1) students actively promote content; 2) promoted content tends to be of 
higher quality than non-promoted content; 3) students use the promotion data to navigate; 4) some 
students are reliably better at assessing quality than others, and they can be identified. Findings like 
these confirm the value of promotions as a learning analytic metric, adding a new dimension to the 
middle space generated by students engaged in loosely coordinated learning activities. The feature is 
simple to use, and easy to implement and understand. 

3 BACKGROUND 

The web has changed the way we do many things, Web 2.0 even more so (O'Reilly, 2007). It all began 
with a proposal on how to manage information more accurately, how to prevent the loss of information 
(Berners-Lee, 1989). This has direct impact on how the web is engineered and how useful it can be for 
education, which is in the business of creating and distributing information. Very soon researchers, 
educators, and science fiction writers started thinking about accessing human "knowledge" with 
notebook-sized computers (Kay, 1991) and creating computer supported (intentional) learning 
environments (CSILE) (Scardamalia & Bereiter, 1994). 


One such learning environment is blogs, which are widely used for various educational purposes (Deng & 
Yuen, 2011; Ducate & Lomicka, 2005; Williams & Jacobs, 2004). Frequently blogs are used as an open 
journal (Zagal & Bruckman, 2007) or a tool of reflection (Deng & Yuen, 2011; Nardi, Schiano, Gumbrecht, 
& Swartz, 2004). Student blogging has been reported to have positive effects on, for example, on the 
following: higher order thinking skills (Ellison & Wu, 2008; Philip & Nicholls, 2009); knowledge sharing 
and reflection (Du & Wagner, 2006; Flong, 2008; Luehmann & MacBride, 2009); the learning process 
beyond the classroom (Ferdig & Trammell, 2004); sense of ownership (Flong, 2008); sense of community 
as measured by the community dimension (Deng & Yuen, 2011); and identity development (Luehmann, 
2008). Some have cautioned that just having students blog does not automatically guarantee positive 
learning outcomes (Divitini, Flaugalokken, & Morken, 2005; Krause, 2004). 


There are two basic types of blogs: a community blog where the whole class posts to a single blog or a 
collection of individual, student-owned blogs. In the case studies presented in this article, each student 
has her individual blog within a single closed community, thus combining the two basic types. When 
viewing the front page of the class, it looks like a community blog where posts by all members are listed 
in reverse chronological order — this creates a sense of community. Alternately, the blogosphere can be 
accessed through the contributions of a single student: each student has her own profile page that lists 
all of her posts — this enables students to develop their identity and feel a sense of individual 
ownership. 


When we say community, we must recognize the importance of the social aspect. Knowledge creation as 
a social product (Scardamalia & Bereiter, 2006; Scardamalia, Bransford, Kozma, & Quellmalz, 2012) is 
complicated and vast; it bounds both formal and informal learning environments as well as networks and 
communities of people. Ferguson and Buckingham Shum propose five categories of social analytics: 1) 
social network analytics, 2) discourse analytics, 3) content analytics, 4) disposition analytics, and 5) 
context analytics. Content analytics is classified as being "a broad heading for various automated 
methods used to examine, index, and filter online media assets for learners" (Ferguson & Buckingham 
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Shum, 2012). In this article, we examine learner-created promotions (i.e., content analytics) and propose 
that they can be used to filter for high quality content in a blogosphere. 


Given the popularity of educational blogs, understanding some of the social features involved will have 
significant value. A knowledge-building community is not simply the text of posts and comments; there 
is so much valuable data for all involved in the learning process. Exploring how many posts students read 
has been found to be informative (Gunnarsson & Alterman, 2012). Meta-data (i.e., promotions, tags, et 
cetera) produced by the students can increase awareness and involvement. Counting comments, 
counting external links, login data, load time, and many other features (as listed in Kargar & Azimzadeh, 
2009) can all help make sense of the online community. 

Blogging is a simple activity requiring little technical skill (click buttons to write, publish, browse, read, or 
comment). Blogs are versatile and widely used for various purposes (Nardi et al., 2004). Different 
methods of measuring content quality have been applied to blogs (Huang, Cheng, & Huang, 2009; Ulicny, 
Baclawski, & Magnus, 2007), such as search and clustering algorithms (Ulicny et al., 2007); voting and 
automated methods based on various quality measures (Kargar & Azimzadeh, 2009); classification of 
different types of blog posts using machine learning techniques (Ni, Xue, Ling, Yu, & Yang, 2007); and 
using natural language processing (Rubin & Liddy, 2006). New features have been added to blogs 
attempting to enhance learning such as dynamic learning maps (Wang, Huang, Jeng, & Wang, 2008) and 
awareness graphs (Ferguson, Buckingham Shum, & Deakin Crick, 2011). 


The blogosphere gives students the opportunity to collaborate outside of class in a loosely coordinated 
fashion (Alterman & Larusson, 2013). Multiple conversations emerge on a single topic, enabling greater 
coverage and diversity with reduced coordination costs. Students publish to the rest of the class, 
providing multiple examples of work in a commons of information (Benkler & Nissenbaum, 2006; 
Bruckman, 1998; Scardamalia & Bereiter, 1994). Each student has her own blog but the collection of 
individual blogs make up the knowledge community. The collaborative effort of the students is not 
jointly focused to solve a particular problem but is rather a loosely coordinated collection of 
representations that together form a collective solution. Multiple conversations emerge from a single 
topic without the need to converge. These conversations persist over the course of the semester giving 
students a variety of resources to reference and refine. 

Blogging enables students to work together, even though they participate at different times from 
different places. Together the students explore the solution space more widely in their individual posts 
but each contribution is examined less closely. In a joint focus space, where students discuss each 
contribution as it is made, that mechanic of working together limits the number of contributions that 
can be considered, but increases the warranty of contributions that survive in the problem space. With a 
loosely coupled learning activity, like blogging, the situation is reversed. Larger numbers of contributions 
become a part of a persistent content space and additional work is required to provide warranties that 
rate the quality of a contribution. Promotions can potentially provide a crowd-sourcing solution to this 
problem: students' subjective assessments of high quality content can provisionally provide a cost- 
effective mechanism for creating warranties and input into a recommender system (Lu et al., 2012). 

4 TWO CASE STUDIES 

The two cases for data analysis come from two classes: the first of 107 students and the second of 50 
students. There was a mix of undergraduate and master's students in both courses. The first class had 
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weekly writing assignments completed in a class blogging environment. There were three kinds of 
assignments: 1) to write an editorial based on a book read in class, 2) to write a review of the book, and 
3) to write a reflection on cross-cutting themes between the assigned readings. Students were 
encouraged to post drafts before the deadline, so they could collaborate as they completed their work 
on each assignment. For this class, students could promote by giving a like or by assigning merit badges 
for various exemplary features of a post. 


We used data collected from the second class (50 students) to confirm the key results of the first study; 
despite a smaller set of promotion types, the key results were verified. In the second class, the students 
had weekly assignments where they practiced the methods and techniques taught in the class. Again, 
students could post drafts and revise their own homework up until the deadline. For this class, there 
were no merit badges, but the students could give anonymous likes to the work of other students. 

5 CASE STUDY ONE: WEEKLY WRITING ASSIGNMENTS IN AN INTERNET 
& SOCIETY COURSE 

The first case study was done in a class on Internet & Society (l&S). During the semester, students read 
four books on the social impact of the Internet and had weekly writing assignments to be completed in 
the blogosphere. Each student had her own blog, composed of multiple posts she authored. Students 
were encouraged to leverage each other's work; to read freely throughout the semester in the 
blogosphere, joining conversations that struck their interest; and to use the blogosphere as a resource to 
improve their own posts. Just browsing in the blogosphere and looking at other students' work has 
tremendous educational value. While working on an assignment, reviewing the posted work of other 
students was allowed. It was also permissible to revise posts, and revise again, up until the deadline. In 
this manner, the blogosphere is a platform for peer tutoring, peer assessment, and collaborative learning 
(Topping, 2005). In addition to the blog assignments, the students also had weekly assignments to 
provide peer comments and assessment "officially" on the posts of other students. 


For each book, the students had to write both an editorial post and a book review post. Each post was 
required to be between 600-750 words in length; in practice, many of the posts were longer than that. 
The editorials required the student to review an issue raised by the book and then take a position, either 
expanding on the argument of the book with examples or presenting counterarguments. Each review 
explicated the central argument of the entire book. Students explained the key points, referenced (at 
least three) editorial posts of fellow students as support for their analysis, provided additional examples, 
and argued for (or against) the core argument of the book. In the latter part of the semester, students 
were required to write three reflections synthesizing material from two or more books and referenced 
posts written on the books. A requirement for these reflections was to reference at least three editorials 
or reviews written by other students. The same comment and assessment process also applied to the 
reflections. 


Each student was required to write two comments and give two peer assessments each week (400 
words in total). After the assignment due date, two posts were randomly assigned to each student on 
which she would officially comment, and another two other posts for which she was responsible for 
giving a peer assessment using a 3-point questionnaire form. The peer comment was posted under the 
student's user name but the peer assessment was anonymous. The comment was public but only the 
author of the post had access to the results of the questionnaire; comments and assessments were to be 
both "thoughtful" and "judicious." To ensure quality, both forms of feedback were evaluated by a grader. 
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Students were also encouraged to make additional comments, respond to comments on their own posts, 
and anonymously promote posts by giving them likes and badges. The additional activities were not 
directly graded, no feedback was given about promotions being good or bad, but promotions were 
considered part of the participation metric in the overall grade. 


Because of the specific requirements of the blogging assignment, the students got plenty of feedback on 
their contributions to the blogosphere. The comments and peer assessments produced by other 
students were one form of feedback. Grades and publicly displayed gold stars for excellent work, given 
by the TAs, were a second form of feedback, and the peer promotions (badges and likes) were a third 
kind of feedback. 


The course assignments were designed to be used as reference material later in the semester. Students 
not only had reason to review their peer work in the current assignment but, because of the design of 
the review and reflection assignments, they also had a reason to review past assignments. This 
potentially meant each assignment was more valuable because it was not just deliver and forget; it had 
future value for the student and the class as a whole. 

5.1 What is the issue? 

Students produced a huge amount of text (600-750 words) on a weekly basis. The official comments on 
the work of two other students after the deadline for the post were a minimum of 200 words. For a class 
of roughly 100 students, this meant 300 contributions per week (one post and two comments per 
student). Or to put it differently, each student was producing 1000 words per week, which meant a 
minimum of 100,000 words per week produced by the entire class. 

Because students were collaborating as they wrote each post, the merits of finding better quality 
material are obvious. High quality content, by definition, is easier to understand, easier to learn from, 
and easier to expand upon. There was also additional value in peer promotions, the results were used to 
gauge the reading and writing ability of the student. Both the reviews and reflections required students 
to cross-reference other posts, meaning that the writer paraphrases or quotes from another student's 
post and provides a hyperlink to it. The reviews were required to refer to at least three editorials. The 
reflections were required to refer to three other posts, either reviews or editorials. Additionally, 
although not required, many students included cross-references in their editorials. 

Two factors contribute to the selection of a post to cross-reference: 1) the student needs to find a post 
relevant to her argument in her own post (the search mechanisms provided by the blogging environment 
supported this activity); 2) even after the initial search, many posts are relevant and useful to a student's 
post — ideally, the student wants to find not just a relevant post, but also a good one. Thus, finding good 
content was important while writing any of the three types of posts. Do student promotions provided 
effective feedback to guide the selection of good content? The results indicate that they do. 

For the instructor, the enormous amount of content being created is problematic for a number reasons. 
The advantage of a student-owned space is that it helps students to open up and explore the material 
among their peers. Ideally, some guidance is provided to help students orient themselves towards the 
better content in the blogosphere, thus improving their overall peer discussion. Given the amount of 
material, depending on the instructor to provide all of this guidance is problematic because the work it 
would entail is enormous. For example, one mechanism for the instructor to provide direction is to give 
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gold stars to the best work for each assignment. The gold stars come after the completion of the 
assignment, providing a standard to which all students in the class can compare themselves. But 
identifying the gold star material in a class of 100+ students is not a trivial task. Can the analysis of the 
peer activity data, the promotions of the students, provide some of this guidance? The results from this 
study say yes. 


5.2 More about the Class: Grading 

There were six graders for the l&S class; this amount of grading support cannot be assumed for a typical 
class. Posts were graded differently than comments and peer assessments. Three graduate students 
graded the posts; a third of the posts were randomly assigned to each of the graders after the due date 
of each homework assignment. The grading was done by filling out a 6-item questionnaire. Each 
homework type (editorial, review, and reflection) had different questions. For example, the grading 
questionnaire for an editorial post was: 


1. The issue is clearly explained. 

2. The opinion is interesting and substantial. 

3. The references are relevant to the argument. 

4. The post is well written. 

5. The post demonstrates understanding of the subject of the book. 

6. Overall grade for this post? 

Each question could be given a value from 0 to 3. Zero means that part was "not completed" (for 
example no references in the post would yield a 0 for that part of the questionnaire), 1 is "not good," 2 is 
"good work," and 3 "exceeds expectations." Students were told to expect a 2 to be a good passing grade. 
Grading was done inside the blogosphere but in a special grading view where only the questionnaire and 
the text of the blog post were visible. The graders could not see the blog post's comments, likes, or 
badges without intentionally browsing to that blog post in the standard view. This "blind review" was 
done to minimize any bias the post's comments and promotions could have on the grader. 


The comments were graded by three undergraduate students. The comment and peer review forms 
were simpler — just asking if the comment or peer review met expectations — the graders could give a 0 
for "not completed," 1 for "not good" and 2 for "good work." Grading did not overlap between graders 
within the same assignment. Posts, however, were randomly assigned to a grader so that over the 
semester each student had assignments graded by all graders. 

Periodically the head TA would give out gold stars to the high quality posts for each assignment. The 
grades of gold star posts varied a bit, although all were of high quality (graded from 14 to 18) some 
provided exceptional examples even if the post as a whole did not warrant the highest possible grade. 
The purpose of the gold stars was to identify high quality content "officially" for the community as a 
whole so that students could compare it to their own. Sometimes the gold stars were given out shortly 
after the assignment due date and at other times much later. A total of 80 gold stars were given 
throughout the semester. 


The average number of person-hours spent grading each week was 31 hours; over the entire semester, 
the total was roughly 350 hours. For each assignment, the instructor would meet weekly with all six 
graders to give an overview of how to grade the most recent assignment. These meetings typically lasted 
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around an hour (totalling ten person hours per week). From interviews with the graduate graders (who 
graded the posts), we learned that the average time spent on grading posts was around four hours per 
grader (twelve hours per week total). The average time spent grading comments and assessments was 
about three hours per grader (nine hours per week total). Clearly, this is an enormous amount of work 
and many courses will not have sufficient resources to commit to this level of grading. Being able to 
simplify some of the process of identifying good content, providing feedback, and evaluating student 
work would have tremendous value. 


5.3 More about the Class: The Technology 

The blogging environment was developed at Brandeis University over a number of years; the current 
version is a recent rebuild by the first author of this paper, which has already been used in several 
classes. We refer to this type of blogging as social blogging because students blog collaboratively, as 
individuals within a group blogging system, and the blog includes several social features, like promotions. 


Students may browse content in the blogosphere using several different views. Most of the time, 
students would start a session by viewing the front page, which lists of the most recent posts. From 
there, a student could switch to an alternate view. Examples of available views include the following: all 
posts on a specific book, all posts by an individual student, all posts with a specific type of promotion, all 
posts receiving a gold star from the TA, and all posts the student had commented upon. It is also possible 
to do a keyword search to retrieve a list of all posts containing the search term. Regardless of the view, 
posts are always listed in reverse chronological order, with the most recently added posts at the top of 
the page. For each post in any view, its entry includes the name of the author of the post, the title of the 
post, the type of post (editorial, review, or reflection), the book(s) the post discussed, the number of 
comments the post had accrued, and the number of promotions it had received. Students could preview 
the first paragraph of a post by hovering over its title; they could then click on the title to view the post 
in its entirety, as well as any comments made on the post. 

5.4 Student Promotions in the Blogosphere 

While browsing in the blogosphere, students could promote posts by assigning them merit badges and 
likes. Liking a post means exactly that — the student supposedly liked it. Merit badges are a more 
specific type of a post promotion. Instead of clicking a button to say you like the post, you can click a 
button to say that you liked it because it was, for example, "nicely written." Students could only give 
each post one like and one badge of each type. Each student could, in practice, give a post one like and 
six different badges. The badges were: 


• Good example: The post contains an interesting example or case as a basis for its argument. 

• Good question: The post raises an interesting issue. 

• Nicely written: The post is well written. 

• Good argument: The post makes a persuasive argument. 

• Good references: The references are interesting and relevant. 

• Good summary: The post provides an accurate and succinct summary of an issue or the book. 

There was incentive to identify high quality posts in the blogosphere — earlier posts were referenced in 
later assignments. By finding high quality posts that clearly explained a topic in one of the books, the 
student could more easily build her argument when reviewing or reflecting on the central issue of the 
book. Reading poor quality posts would not be as helpful. Searching for good quality posts takes work, 
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but also depends on a student's ability to identify what is a good post. The data will show that not all 
students were good at assessing quality. 


5.5 Evaluation: The Data 


The last assignment of the semester is left out of the data, as it is an outlier. Student participation in the 
last assignment was very different from the previous ten. This is most likely due to the fact that the 
semester was ending and students were busy finishing up their work in many classes. 


The data compiles the promotion statistics of 92 students. The 15 students filtered out did not 
participate fully in the blogosphere and there were several possible reasons for excluding their data: only 
completing the first one or two of the 11 assignments, incomplete assignments, not participating in the 
blogosphere, or finishing most of the work after the end of the semester. For each of these students, 
their data was removed from the set. 

5.6 Evaluation: Overview 


All of the posts were graded, during the semester, on a scale of 0 to 18. For the purposes of this 
evaluation, posts with a score of 14 to 18 are high quality posts, 9 to 13 are average quality posts, and 
below 9 are poor. An outline of the evidence presented in the evaluation is shown in the Summary of 
Evaluation below. The utility of student promotions depends on several factors. For this scheme to work, 
students must actively promote during the semester, and the content they promote must in general be 
of high quality (evaluations 1&2). This information is useful for students only if they use promotions to 
help navigate in the blogosphere (evaluation 3). If promotions can be used to identify better content, 
they can potentially be used for highlighting content — as support or replacement for the identification 
of gold star material — and therefore, improving the reliability of promotions can greatly improve their 
functionality. Flow can this be done? One way is to separate the good promoters from the bad 
promoters. The evaluation presented explores whether promoters whose evaluations are reliable can be 
differentiated from those whose evaluations are more random (evaluation 4). The evaluation also 
explores how promotions and the content quality changes during the semester (evaluation 5). 


5.6.1 Summary of Evaluation 

Evaluation 1. Do students promote posts as they read in the blogosphere? 

Evaluation 2. Are the promoted posts, on average, of higher quality than the average blogosphere 
post? 


Evaluation 3. Do students use promotions to navigate in the blogosphere? 

Evaluation 4. Are some students more reliable evaluators of quality content than others? 
Evaluation 5. Promotions and post quality during the semester. 


The data shows that students produce many promotions, many posts do not receive promotions, and 
the posts that receive promotions vary in how many promotions they receive. The average grade of 
promoted posts is significantly (p-value < 2.2e-16) higher than that of non-promoted posts. Some 
students were better than others at promoting, and as the semester progressed, the quality of the 
content in the blogosphere increased. 

5.7 Evaluation 1: Promoting Content 

Under what conditions are student promotions useful? If the promotion feature is rarely used, they risk 
getting lost. If there promotions are too many, they become meaningless. For the entire class, 89.7% of 
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students used the promotion feature at least once. In the first assignment, many students used the 
feature, but the number of students who promoted declined in later assignments. During the whole 
semester, the average student promoted 6.77 posts from 3.75 assignments. For each assignment, 37.8% 
(an even third if the first assignment is excluded) of the students gave out a promotion. During the first 
assignment 78.2% of students gave a promotion and during other assignments between 50% and 16.3% 
(in the last assignment) gave promotions. 


The proportion of the number of promotions per assignment follows a similar trend, each of the ten 
assignments received, on average, 10% (8.5% excluding the first assignment) of the total number of 
promotions given throughout the semester. The first assignment received 23.7% of all promotions with 
other assignments getting between 11.9% and 4.9%. 

Around 80% of promotions were made before the official commenting period ended (and thus before 
the assignment was graded). This makes them very useful while students are working on the same 
assignment and helpful when students are searching for quality content to reference in later 
assignments. Quite a few promotions were available before the assignment deadline, giving students 
access to higher quality content, on average, to help understand the problem. Later, when reviewing 
prior assignments, the search function can be used to find relevant content and then the promotions 
quickly indicate the best posts. 


Later assignments were less likely to be used as reference material because they were designed as 
reflections on the first eight assignments. Students are also busy with exams at the end of the semester. 
Both of these aspects could have a reducing effect on promotions. Despite the outliers, the first and last 
assignments, we believe both the rate and number of promotions given throughout the semester were 
good. 

5.8 Evaluation 2: Quality of Promoted Content 

This evaluation seeks to answer the core question of this article: Does the promotion activity identify 
high quality with minimal instructor involvement? In order to answer this question we ask what was the 
quality, in terms of grades, of the posts that received a promotion? We answer this question in three 
parts: 

• Higher quality posts got more promotions. (See: Number of promotions per post) 

• Higher quality posts were more likely to be promoted. (See: Promotion hit rate) 

• All types of promotions were useful (See: Different types of promotions) 

If higher quality posts get more promotions and if higher quality posts are more likely to be promoted, 
then the subset of promoted posts is of higher quality. How much we answer in the following 
evaluations. 

5.8.1 Number of promotions per post 

Figure 1 shows the total number of promoted posts and the total number of posts not promoted, both 
grouped by grade. It also shows the number of total promotions for posts of each grade. The data shows 
an interesting increase in the number of promotions on content with high grades (see the shaded region 
in Figure 1). In other words, all posts can receive promotions, but posts with high grades get significantly 
more promotions than posts with a lower grade (p-value = 0.002045) (see Figure 2). 
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Figure 1. Number of posts and promotions per grade 
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Figure 2. Average number of promotions per grade 
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5.8.2 Promotion hit rate 

Were high quality posts more likely to be promoted than lower quality posts? Only 36.2% of all low 
quality posts (graded 8 or lower) were promoted. This means low quality posts have a 36.2% hit rate. 
The hit rate increased to 47.3% for posts of average grade (graded between 9 and 13), and high quality 
posts (graded 14 or higher) had a hit rate of 54.3%. The difference in hit rate average is only significant 
between poor and high (p-value = 0.033) but the distinction between poor and average is higher (p-value 
0.11) than between average and high (p-value = 0.2165); students seem to identify average posts over 
poor posts more easily than high quality posts over average quality posts. 


Comparing the average grade of posts with and without a promotion paints a clearer picture. The 
distribution of promotions between high and low quality posts is significantly different (see below), 
indicating that students can, on average, identify high quality peer content. 

5.8.3 Different types of promotions 

As mentioned before, we named the activity of liking and giving a badge as promoting. We analyzed the 
data for each type of promotion (see Figure 3). The average grade for all posts during the semester was 
10.7 (see horizontal line for average class grade). With one exception, all types of promoted content had 
higher grades than the average class grade. 

There were six different types of promoted posts in the blogosphere: 1) posts that were both liked and 
badged; 2) posts that were liked and could have badges too, 3) posts that were liked and had no badges, 
4) posts that got badged and could also have likes; 5) posts that were only badged and had no likes; and 
6) posts that received no likes or badges. Comparing the five groups that had promotions to the group 
(6) that had no promotions: 

1. Posts that had 1+ likes and 1+ badges had a significantly higher grade (p-value < 2.2e-16). 

2. Posts that had 1+ likes and 0+ badges had a significantly higher grade (p-value < 2.2e-16). 

3. Posts that had 1+ likes and 0 badges had significantly higher grades (p-value = 8.181e-09). 

4. Posts that had 0+ likes and 1+ badges had significantly higher grades (p-value = 0.006887). 

5. Posts that 0 likes and +1 badges did not have significantly higher grades (p-value = 0.3264). 
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Figure 3. Average grade per type of promotion 
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Badges do not add anything to the predictive power of likes, statistically. The numbers, however, are 
different for each badge, as shown in Figure 3. Badges can prime posts for likes or provide additional 
meta-information of value. 


5.8.4 Summary 

Combining all of these results (keeping in mind that 80% of all promotions happen before assignment 
deadline) — high quality posts get more promotions, the promotion hit rate is higher for quality content, 
and all types of promotions are useful — this means promotions are a powerful method for identifying 
quality material: students can successfully identify high quality content. 

5.9 Evaluation 3: Navigating Promotions 

On average, each post was read 22.9 times during the semester; gold star posts were read, on average, 
44 times. The promoted posts were, on average, of higher quality, but were the students guided by the 
promotions to read those posts more frequently than the average post? Posts without any promotion 
feedback were read 16.8 times. Promoted posts got 33.5 reads — twice as many! Not only are students 
promoting quality content but they also act on the promotions. The community as a whole steers itself 
towards quality content using the promotions feature. 
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Figure 4. Average number of reads before and after promotion. 

Some posts only get likes, some posts only get badges, other posts (25 out of 80 total gold stars) get only 
gold stars, and finally many posts get no promotions or gold stars. All posts get about the same average 
number of reads (~15) before any promotions or gold stars. After a post gets a promotion its' exposure is 
greatly increased and it accrues a lot more reads from students (see Figure 4). Gold stars are given after 
the deadline, shortly after grading (usually about a week or two after deadline), and do increase the 
number of reads a post receives. Why a gold star only post has more reads before it gets a gold star is 
interesting, a possible reason is because they are cited in other students' posts (see cross-links below). 
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These posts also get fewer average reads than promoted posts but that is most likely because gold stars 
are awarded later. 


How students use promotions can be seen in students' cross-links (links to other students' posts within 
their own post). On average a cross-linked post has 1.93 promotions whereas a post that was not cross- 
linked only has 0.88 promotions, on average. The quality, measured by grade, of the cross-linked posts is 
also significantly higher (p-value = 4.778e-12). Students clearly use the promotions to navigate and use 
them as a source to improve their own work. 

5.10 Evaluation 4: Promotion Predictions 

Are some students better at promoting quality material than others? We wanted to know which 
students were reliable: both among students who consistently promoted high quality content and 
students who tended to promote poor content. Figure 5 shows that some students promote high quality 
posts, other students mostly promote low quality posts, but the majority of students are less 
predictable. A third (33) of the students regularly promote posts (5 or more assignments); a quarter (23) 
of all students consistently promote high quality posts (labelled good promoters in Figure 5). Out of 
those 23, seven students are also regular promoters, averaging 16.7 promotions throughout all 
assignments. 

In Figure 5, good promoters refers to students who on average promoted posts that received a good 
grade (on or above the top line), and poor promoters are the students who promoted posts that 
received a poor grade (on or below the bottom line). Obviously, the posts promoted by students who 
reliably promote high quality material can be used to feature content in the blogosphere. In terms of 
assessment, the users classified as poor promoters present an interesting case. First of all, getting them 
to stop promoting content will immediately increase the average quality of promoted content. Secondly, 
this information can be used by the instructor as a teaching opportunity — these users lack an important 
critical reading skill. 
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The distribution is interesting in that the grades of students and the grades of the posts they promote do 
not correlate. One might expect that students who produce high quality content to be the same ones 
who can identify quality content by promoting. However, some students who get high grades only 
promote average or low quality content. Perhaps, even more surprisingly, there are students who receive 
low grades on the posts they produce but are very good at identifying and promoting high quality 
content. This suggests that students are developing two different skills (that could be classified by 
drawing a line up and along the average grade (10.7) on both axes): 1) writing and 2) identifying quality 
content. A student can be good at both, either, or neither. One of the benefits of the promotion data for 
instructors is that it can be used to measure the writing and critical reading abilities of each student. We 
consider this an important finding, one that can make promotions a very useful feature in educational 
blogs. 


5.11 Evaluation 5: Promotions and Post Quality During the Semester 

The distribution of grades and accuracy of promotions did not stay the same throughout the semester. 
The number of good predictors increased slightly and the number of poor predictors decreased by more 
than half. At the same time, the number of poor quality posts decreased and the number of average 
posts increased. What this tells us is that the average quality of the content is increasing and that the 
students are becoming more accurate at promoting. Analysis of the data is not that conclusive though. 


A three-factor ANOVA test of assignment number, likes, and badges indicated a negative trend in student 
promotions. While there was a slight increase in mean assignment grade over the course of the 
semester, it was not significant (F = 0.983, p-value = 0.452). There was a high negative correlation in the 
difference of marginal means (Pearson correlation -0.8, p-value 0.005). This can be explained by the 
change in the distribution of grades (see Figure 6), making it more difficult to identify high quality posts. 



Assignment 


■ Grades: 0-8 

■ Grades: 9-13 

■ Grades: 14-18 


Figure 6. Number of posts per grade and assignment. 
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Remember from the grading rubric, a grade of 12 was considered a good post; high quality posts are 
considered to "exceed expectations." During later assignments, having a much higher number of average 
quality posts compared to poor quality posts means that it is much harder to differentiate between high 
quality and poor quality. The main difference between the group of students who promote high quality 
and poor quality is the reliability of good promoters. Figure 7 shows the number of people who could be 
called "consistent promoters"; at least half of their promotions were either of high quality or poor 
quality. 

Predictors per assignment 



- Good promoters 

- Poor promoters 


Figure 7. Number of consistent promoters. 


Ideally, students improve their promotion skills over time, meaning that as the semester progresses 
students become even more accurate at identifying high quality content. Unfortunately, the results are 
mixed for this claim. The number of promotions went down over the semester while the overall quality 
of content went up. Students became less accurate with their promotions but their hit rate was slightly 
better. 

There was considerably less poor content in later assignments and fewer students consistently giving 
poor promotions. However, the number of high quality posts was about the same as poor quality posts 
in later assignments, yet the number of reliably good promoters remained steady and exceeded the 
number of people who promoted poor posts in later assignments, even though the number of good and 
poor posts was about the same. This is an echo of the hit rate in evaluation 2: higher quality content has 
a higher promotion hit rate than lower quality content. 
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In an environment of contrasting content, in terms of quality, differentiation is easy. Exploring the hit 
rate between assignments provides an insight. During the promotion-stable part of the semester 
(assignments 2 through 9), there was a noticeable drop in the promotion hit rate for poor and average 
quality posts between the first half of the assignments and the second half. The poor quality hit rate 
dropped from ~35% to ~30% and the average quality hit rate dropped from ~50% to just below 45%. The 
high quality hit rate was the same at around 56%. While the overall quality of content improved, the hit 
rate of lower quality posts declined. 


Early in the semester, the difference between high quality posts and poor quality posts was much larger 
than later in the semester. Notice in Figure 6, because of the pattern of assignment (editorial, review, 
editorial, review, et cetera), the improvement began to emerge with the second assignment of each 
type. With the overall improvement in the quality of posts, the discrimination between high quality and 
poor quality becomes more difficult to make, but also less significant; in other words, the average 
promoted post was always of higher quality than one that was not promoted. The change in overall 
quality is a good thing for the students, because reading better material is more productive. The goal of 
having students read better content is achieved with the overall improvement in content and by 
providing a smaller subset of higher quality posts through the promotion feature. 

5.12 Summary of Analysis 

Evaluation 1. Students promote posts as they read in the blogosphere. 

Evaluation 2. Students accurately identify high quality content. 

Evaluation 3. Students use promotions to navigate. 

Evaluation 4. Students can be classified by the content they promote. 

Evaluation 5. As the semester progresses, students have access to better content. 


So, does this mean that the promotion activity identifies high quality with minimal instructor 
involvement? Because students actively promote, because the promotions are useful, and because we 
can identify good predictors and poor predictors, high quality content can be predicted based on who 
promotes it. Yes, the content students promote is of considerably higher quality than posts not 
promoted. In addition, the analysis provides hints that the promotion feature can, to some extent, be 
used to measure a student's critical reading and writing skills. 


6 CASE STUDY TWO: WEEKLY ASSIGNMENTS IN A HUMAN COMPUTER 
INTERACTION COURSE 

Data from another case study confirms some of the results of the initial study concerning the quality of 
promotions; that students actively use promotions and that they accurately identify high quality content. 

The basis of the second study was a course on Human Computer Interaction (HCI) taught in a Computer 
Science department and cross listed in Psychology (Alterman & Gunnarsson, 2013). There was a mix of 
50 undergraduate and master's students in the course, with weekly homework assignments. Homework 
was done in a blogging environment. The weekly assignments were blog posts where students either 
implemented or applied HCI techniques to various assignments. Each week there was an assignment 
deadline and a comment deadline. Students could, at any time, read, comment, and give likes. For this 
class, there were no merit badges. 
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Students were free to post drafts of their homework before the deadline, read each other's drafts, and 
revise their assignments up until the deadline. After the completion of each assignment, there was an 
official three-day commenting period; students were required to post substantial comments on the 
randomly assigned posts of two other students. As the semester progressed, some of the earlier work of 
the students became relevant to the term project. Thus students were to free browse, comment, and 
like the work of other students. The data shows that many students actively read in the blogosphere 
throughout the semester (Alterman & Gunnarsson, 2013); they regularly read each other's posts while 
working on each assignment and continued to read them after the completion of each assignment. The 
term project was purposefully designed in such a way that the content created in previous assignments 
could be used as source material. While writing the term project, students frequently browsed previous 
posts. 


The same blogging environment was used in both classes, the l&S class had been upgraded to use 
badges and some minor Ul (user interface) changes had been made. For the HCI class, the front page 
listed the most recent posts in reverse chronological order. The students could filter the posts by 
assignment, by author, or by using a search mechanism. Regardless of how the posts were filtered, the 
list of posts generated was always in order of the most recent, with each post summary including a count 
of the number of likes and comments. The content of a post could be previewed by hovering with the 
mouse over the title. 


Grading of posts and comments was done by three graduate students. The grading scheme was simpler 
than that employed for the l&S class. Where for the l&S course the parts of the post were graded, for 
this class a single grade, on the scale of 0-3 was given to each assignment: 

• 0 means that part was "not completed" 

• 1 means "not good" 

• 2 means "good work" 

• 3 means "exceeds expectations" 

Comments were graded on the 0-2 scale where the numbers meant the same as for post grades but 
comments could not exceed expectations. Each post was graded by a single grader; posts were randomly 
assigned to each grader for each assignment. The instructor and the graders had weekly meetings, 
before the grading for an assignment began. During the meeting, several assignments were graded 
collectively to coordinate the grading rubric for the assignment. 

Students actively used the promotion feature; the average quality of liked posts was higher than posts 
that were not promoted (see Figure 8, which shows grades scaled between 0 and 1). 

Table 1 compares some of the key numbers in this second study to those in the l&S study. The students 
in the l&S class did more promoting, but those promotions also include badges; liking an entire post is a 
more stringent criteria than liking some feature of the post, for example, if it had "good references." 
While a student might not be ready to give a post a like, she might consider giving the post a badge. For 
both classes, high quality material was more likely to be promoted than poor quality material, as 
measured by the hit rate. Some differences between hit rate numbers are undoubtedly a result of the 
difference between the grading and promotion schemes in the two classes. To repeat: in the HCI study, 
students could only "like," there were no merit badges, and grades were on a simpler scale, from 0 to 3. 
The merit badges in the l&S course were less accurate predictors of high quality content than the likes 
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but they do contribute to a greater hit rate of high quality content. The simpler grading scheme and the 
lack of merit badges probably explain the lower hit rate of poor quality posts in the HCI class data set. So 
while the high quality hit rate in the HCI study might not look as impressive as in the l&S one, the low 
poor quality hit rate is impressive — it makes the subset of promoted posts less "polluted" by poor 
quality content than the promoted l&S content. 

Average grade difference 


■o 

E 

O 



Liked average Not liked average 


Posts 


Figure 8. Difference between the average grade of posts that were "liked" and those that were not. 

Grades are scaled between 0 and 1. 


Table 1. Comparing data from the two case studies. 



l&S 

HCI 

Promotions per user 

6.77 

4.1 

High quality hit rate 

54.3% 

45.5% 

Poor quality hit rate 

36.2% 

18% 


To summarize, the data from the second case study confirms some of the key results of the l&S case 
study: 

1. The students actively promoted content throughout the semester. 

2. The promoted posts were, on average, of higher quality than the posts that were not promoted. 

3. High quality content was more likely to be promoted and thus had a higher hit rate. 
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In our class, students get feedback through grades, comments, peer assessments, and promotions. 
Future work will evaluate the effectiveness of the promotion activity without the rigorous grading done 
in this class. If instructors were to grade fewer posts, would student promotion improve over the course 
of the semester as the data has shown? What if the only posts graded were the promoted ones? 


Consider, as a possible grading scheme, for the instructor to grade only promoted posts. This reduces 
work for the instructor. Under this grading scheme, it would be possible to ascertain which students are 
good predictors, which students are less predictable, and which students do not participate. This level of 
effort on the instructor's part is sufficient to develop some kind of highlighting mechanism, possibly a 
replacement for the gold star feature. The obvious downside of this scheme is that some students would 
not be receiving feedback or grades from the instructor (or the graders). If needed, posts that were 
never promoted could be randomly sampled for grading purposes. With this additional work, the 
instructor would be able to monitor the progress of students from assignment to assignment, both as 
readers and contributors. Regarding feedback, students would be receiving regular feedback from the 
instructor (but not every week). They would also be receiving peer feedback in the form of comments 
and promotions, and have access to highlighted content to compare to their own work or to build upon. 
With this scheme, some of the posts that were not promoted but were of high quality would be 
discovered. 


In a blogosphere, as a learning environment without the promotions described in this article, there is a 
possible bias for work posted either early or late. Do early posts garner disproportionately more 
attention and feedback? Do poor quality early posts have a negative impact on the discussion? Do later 
contributions suffer from the opposite problem: too little feedback and promotions from peers? Is the 
peer assessment requirement the driving force of promotions, without which there wouldn't be many 
promotions? The hope is that the quality of contributions gradually increases towards the due date as 
students read and update previous contributions based on more recent and collected content, and that 
the early post bias does not slow down this progress. 

The blogosphere, with the promotions described in this article, potentially compounds the early and late 
bias. Based on the results, the promotions may have a positive impact on the early post bias by 
identifying the high quality early posts, hinting that those not promoted may need updating. In the early 
bias case, promotions may be the solution. The problem in the other direction — overlooking good 
quality posts contributed late — may require some scheming to remedy. One possibility is to leverage 
the random assignment of required commenting. During the commenting period students can, and do, 
promote content. If these late promotions are made by good promoters then they become prime 
candidates for further review as gold star contributions. But in either direction, the question of bias 
requires further study. 

Suppose the blogosphere has only a few users that produce little content. Would promotions work 
differently with this number of students versus this content-amount dynamic? Since there is so little 
content, students who want to collaborate or need help can just read everything. Also, would a more 
intimate setting of only a few users trigger everyone to promote everything? In this case, the promotions 
are working differently as mostly a positive feedback mechanism supporting student engagement. 
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One part of the class was peer assessments. Generally, students did not give likes to the posts they were 
assigned to assess (only 10 likes from seven students); students were more likely, however, to give 
badges during assessments (58 different students on 213 occasions giving badges to 196 different posts). 
This amounts to just fewer than 6% of all likes and 21% of all badges awarded. Students only awarded a 
post with a promotion if they had assessed it to be of high quality; however, a brief exploration of the 
peer assessments indicated that they were not reliable. The interaction between peer assessments and 
promotions could be a topic of another analysis. For this article, assessments may have had a negative 
impact on the quality of badges, the difference in average grade between badges awarded during 
assessment and those awarded at other times was significant (p-value = 0.02434). The difference in 
means, while statistical, is only half a grade point. We cannot conclude that, without the assessments, 
badges would have been of a higher quality. 


While badges do not add much statistical value to likes, they are numerous and as such of social value. 
Would likes be as predictive, for example, without badges? The difference in grade schemes in the HCI 
and l&S courses makes it difficult to say. The badges may add a level of assessment between a like and 
no like, thus making the difference between a like and no like greater than it would be without badges. 


Do these results generalize beyond blogging? Suppose the students are tweeting (many contributions 
from either many or a few students). Does the notion of promoting make sense under these conditions? 
Perhaps, but for this kind of learning activity the middle space would be somewhat different, and 
consequently the analytics would clearly have to be adjusted. For various kinds of loosely coordinated 
learning activities, promotions hold promise, but the devil is in the details. Whether and how to convert 
the promotions into assessment and feedback would necessitate further study. 


Overall, the results of this research are very promising but we do not assume that the promotion feature 
can be automatically applied to any learning environment and be a success, as explained by the middle 
space. Flowever, the possibilities the promotion feature enables for learning analytics are of interest. 

8 CONCLUSION 

In this article we have explored the use of peer promotions as a tool to filter for high quality content. The 
data shows that students willingly use the tool and that the application of the tool provides the desired 
results — the promoted content is of significantly higher quality than content that is not promoted, and 
content that is repeatedly promoted is of higher quality than content that has fewer promotions. These 
results have been verified by two different case studies. Other results show that good and poor 
promoters can be identified. Both classifications of promoters have value: by focusing on good 
promoters, the reliability of quality assessment can be improved; by focusing on poor promoters, the 
instructor is in a better position to identify students who may be struggling. 


Navigating a huge amount of peer-generated content in a knowledge community on a weekly basis is a 
lot of work. Not all content is of sufficient quality to be considered useful for all students. Being able to 
filter the content for quality would be invaluable. The social activity of promoting each other's material, 
if properly applied, enables students collectively to identify content that is interesting or of particular 
value. What the number of promotions might mean for a post will vary from one setting to another, but 
the promotion feature will always have social value. 
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